Reversibility, heat dissipation and the importance of the thermal 
environment in stochastic models of nonequilibrium steady states 
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We examine stochastic processes that are used to model nonequilibrium processes 
(e.g, pulling RNA or dragging colloids) and so deliberately violate detailed balance. 
We argue that by combining an information-theoretic measure of irreversibility with 
nonequilibrium work theorems, the thermal physics implied by abstract dynamics 
can be determined. This measure is bounded above by thermodynamic entropy 
production and so may quantify how well a stochastic dynamics models reality. We 
also use our findings to critique various modeling approaches and notions arising in 
steady-state thermodynamics. 

A theory of nonequilibrium physics is vital if we are to understand such diverse phenom- 
ena as geological or biological processes which are inherently dissipative in nature. Although 
a general theory remains both challenging and elusive, it is now possible to obtain precise 
experimental data for mesoscopic objects such as RNA strands [l| and optically-trapped 
colloids [2I undergoing irreversible manipulation. In turn this has allowed theoretical devel- 
opments, such as strikingly general nonequilibrium work relations, to be verified [ij]. 

In this work, we address the fundamental question of how to faithfully model irreversible, 
dissipative physics with stochastic dynamics. We introduce an irreversibility measure for 
stochastic processes which, in contrast to standard expressions, respects such basic physics as 
frame invariance. We find that an explicit prescription for a system's thermal environment— 
often absent in models — is essential if predictions for entropy production are even to be 
possible. Using work relations for stochastic systems 0] we find our main result, inequality 
(JH) below, which shows that such predictions always underestimate the true dissipation, 
unless all relevant processes are modeled. This suggests that a model predicting less dissi- 
pation than is observed is incomplete; and one predicting more should be rejected. Since 
our results hold for arbitrary nonequilibrium states, we gain many insights into theories and 
models of nonequilibrium steady states (NESS) [5( which cannot not be drawn from, for 
example, a similar expression recently derived for isolated systems constrained initially to 
be at equilibrium jfjj]. 

We begin by reviewing the modeling paradigm introduced by Katz, Lebowitz and Spohn 
in their seminal work on fast ionic conductors [7]. One takes the master equation for 
a (discrete-time) process, P t +\{C) = J2c> Pt(C')M(C — > C) in which Pt(C) is the time- 
dependent distribution of microstates C. For a reversible, equilibrium system, the transition 
probabilities M(C — > C) are taken to satisfy the detailed balance condition 

P*(C)M(C -> C) = P*(C')M(C -> C) (1) 



FIG. 1: Biased diffusion arising from the local/generalized detailed balance principles applied to 
hard-core particles in a one-dimensional linear potential gradient under (a) open and (b) periodic 
boundary conditions. 

with respect to the Boltzmann distribution P*(C) = e~^ E ^ where (3 is inverse temperature 
and E the internal energy. This relation guarantees microscopic reversibility jsl, 0] in the 
steady state — i.e., that any sequence of configurations is witnessed with the same probability 
as its time reversal. To model irreversible physics, and in particular a NESS, one must 
deliberately violate detailed balance. 

There is no obviously correct way to go about this, so following [3] it is commonplace to 
invoke a local (or generalized [13]) detailed balance principle. In this approach, ([!]) is taken 
to apply over some closed subset of configurations, and a nonequilibrium system formed 
by joining together subsystems that are in contact with heat baths at different tempera- 
tures. For illustrative purposes, we take the specific example of hard-core particles in a 
one-dimensional linear potential, which if connected to particle reservoirs at different densi- 
ties, would exhibit the biased diffusion shown in Fig. UK Alternatively, periodic boundary 
conditions might be imposed (Fig. CEJd), at the expense of being able to couch the dynamics 
in terms of a single-valued potential. Note, however, this modeling procedure can be used 
for all types of particle interaction and in any dimension. 

A problem with this approach is that one loses sight of how the system interacts thermally 
and mechanically with its environment, and could thus be argued to lack a firm physical 
basis. Furthermore, it is not obvious that alternative approaches, e.g., those based on 



maximal entropy analyses subject to macroscopic flux constraints [lll.ll2j. offer more realistic 
descriptions of nonequilibrium physics than the model-building tradition described above. 
We address these shortcomings by introducing a framework in which a model system's 
thermal environment is made completely explicit, which, as we now show, is necessary to 
establish the degree to which a stochastic dynamics is irreversible. 

The standard way to do this is to compare the left- and right-hand sides of the detailed 
balance condition (JTJ). For example, logarithms of their ratio appear in a widely-used ex- 



pression for the entropy production attributed to Schnackenberg 13], an action functional 



ihat exhibits a Gallavotti-Cohen symmet ry 14j and various time-dependent generalizations 
il 15], as well as the housekeeping heat 16] that is instantaneously dissipated b y a NESS 



since 



171 ]. Their differences, meanwhile, have been proposed to characterize a NESS [18 
instantaneous physical currents (which vanish at equilibrium) can be derived from them. 
The violation of ([T]) is thus almost universally used to recognize a dynamics with a dissipa- 
tive steady state, despite the obvious shortcoming that an observer can witness the former 



FIG. 2: Comparison of trajectories generated by the forward and reverse processes. Forward 
trajectories of length T are generated, at which point all velocities (shown as short arrows) are 
flipped, and the reverse dynamics started. The heaviness of the lines indicates the probability of 
the trajectories in each ensemble. The central trajectory appears with the same probability in 
both ensembles, whereas the outer trajectories appear with different probabilities, thus indicating 
an irreversible dynamics. 



without the latter simply by changing frame. 

This difficulty is resolved by realizing that when comparing the probability of a trajectory 
with that of its time reversal, the latter should not be drawn from the same ensemble as 
the former, but from an ensemble in which all degrees of freedom in the environment are 
also time reversed. The dynamics that generate this second ensemble we shall call the 
reverse process. We may now define the following general measure of reversibility for any 
stochastic dynamics, i.e., not restricting ourselves to ergodic time-homogeneous Markov 
chains in discrete time (see also Fig. [2]). Let X denote a trajectory (C 1; C 2 , . . . , C n ) that 
visits configuration Ci at time tj, possibly other (unspecified) configurations at other times 
and eventually reaches, with probability Pt(Ct), configuration Ct at a time T > t n . Given 
an initial configuration Co that is drawn from a distribution Pq(Cq), this trajectory is taken 
to appear under the forward dynamics with probability P(X\Cq)- This is to be compared 
with the probability of seeing the time-reversed trajectory X, in which the image Ci of Ci 
under time reversal (i.e., with all velocities reversed) is seen at time ti running backwards 
from time T to 0. Given a starting configuration Ct drawn from a distribution Pt(Ct), this 
reverse trajectory appears with probability P(X\Ct)- If there is to be any possibility for the 
ensembles of forward and reverse trajectories to coincide, we must take Pt(Ct) = Pt{Ct), 
i.e., start the reverse process by immediate time reversal of configurations reached after time 
T under the forward dynamics. Any other choice requires us to make additional assumptions 
on the dynamics. 

In the spirit of Landauer's principle 19j, we now loosely associate information lost under 
the dynamics — quantified here by the additional information required to reconstruct the 
forward trajectory ensemble from the reverse — with irreversibility and dissipation. This 
amount of information (in natural units) is given by relative entropy of the two ensembles 
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A/= £ Wf (^) h ™^ . (2) 
c ,x,c T P T {L T )P{X\L T ) 



To make contact with thermal physics, we assume that just before the start of the forward 
and reverse processes, any heat baths present are manipulated by a thermostat in such 
a way that the probability that any particular bath configuration is realized is given by 
the Boltzmann distribution with a well-defined temperature. Note that this necessarily 
requires correlations between the system and bath to vanish rapidly — this is the origin of 
dissipation, as will be_seen concretely below. Under such conditions, Jarzynski's detailed 
fluctuation relation 
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In P (^ A ^v|Co) = AS env (3) 

P(X , — AS^ \Ct) 

applies (in a system of units where Boltzmann's constant is unity). Here, AS cnv is the total 
entropy increase in the heat baths under the forward dynamics. This is a random variable 
if the trajectory X contains insufficient detail to determine how much energy has been 
exchanged with each heat bath separately. In deriving this formula, it was assumed that the 
microscopic evolution is Hamiltonian with respect to potentials that are time-independent 
in the baths but may exhibit time dependence in the system of interest. Stochasticity 
enters from the Boltzmann sampling of bath configurations and any coarse-graining in the 
specification of the trajectory X . 

We finally arrive at an important inequality — the main result of this work — by averaging 
over AS^nv An application of the log-sum inequality 



(which itself is a consequence of Jensen's inequality |20|) leads to 



< AI < (AS e nv) + S G (T) - S G (0) (4) 

in which Sc{t) = — Yle Pt(C) ^Pt(C), the Gibbs entropy of the distribution at time t under 
the forward dynamics. Whilst a similar result was recently given for isolated systems starting 
at equilibrium [6], our result (jll) holds for any initial condition and explicitly requires the 
system to be open to the environment. Moreover, the thermostatting of the baths means 



that S env is the true entropy production, which is not always true of isolated systems 22] . As 
we now discuss, (jl]) thus provides hitherto unavailable information — spatial and temporal — 
about heat production in a general nonequilibrium system, e.g., a NESS. 

For example, the lower bound is attained only if every forward trajectory appears with 
the same probability as its time- reversal in the reverse ensemble (i.e., no information loss 
occurs and the process is reversible). This leads to an extended detailed balance relation for 
a NESS, viz, 

P*(C)M(C -> C) = P*(C')M(C' -> C) . (5) 

Note that one cannot decide on the reversibility of a dynamics until its reversal M(C — > C) 
has been identified (see below for concrete examples). Since equality of forward and reverse 
trajectory sets implies P*(C) = P*(C), one finds (jSJ) can be written in a more symmetric 
form with P*(C) on the right-hand side. This condition is equivalent to (El), as can be shown 



from conservation of probability J2c M(C — > C) = 1. The condition ((Sj) can also be stated 
Kolmoeorov criterion Igl 

M(Ci -> C 2 )M(C 2 -»• C 3 ) • • • M(C T -> Ci) = M(Ci -> C T )M{C T -> dr-i) • • • M(C 2 -> C x ) 

(6) 

on every loop in configuration space of length T > 1 . This allows reversibility to be decided 
without prior knowledge of the stationary distributions P*(C) or P*{C). Equivalence of (jSJ) 
and (jSJ) is shown in a similar way to the standard case jl]. 

The upper bound in (J4]) is reached only if the stochastic dynamics faithfully models all 
dissipative processes in the physical system. This we have already seen from the fact that if 
one cannot work out from the trajectory X how much energy has been exchanged with each 
bath, AS'env in (jSJ) is a random variable and AI underestimates the true entropy change. 
As for isolated systems starting at equilibrium jfj|, the log-sum inequility implies that AI 
further decreases under spatial coarse-graining. Since in the present, more general context, 
AI contains temporal information, coarse-graining in time, or reduction of a non-Markov 
dynamics to a Markov process, has the same effect. We thus suggest that this reduction in 
AI could reveal the amount of heat dissipated at the finer-grained scale, and that differences 
between a model's prediction for entropy production and that measured in a real system 
might allow deficiencies in the model to be identified. 

Finally, we use relation (j3J) to gain new insights into the stochastic modeling approaches 
described earlier. The physics of the open system (constructed using generalized detailed 
balance [lo| . Fig. [T^,) is consistent with that described above. In particular, the interpre- 
tation of \nM(C — > C')/M(C — > C) as being proportional to the energy exchanged with a 
reservoir holds, as long as one is confident that all dissipative processes are captured by the 
Markov dynamics of particles hopping on a lattice, and further that one can unambiguously 
identify which bath exchanges energy at any given transition in the system of interest. Note 
that since particle velocities are not included in the model C = C; also M = M as the po- 
tential is time-independent. We also see explicitly that dissipation results from a continuous 
thermostatting of the reservoirs that enables particles to enter or leave the system with a 
constant probability in every timestep. 

Models in which a current is induced by periodic boundary conditions (see Fig. [Tb) are 
more subtle. There are at least two ways in which such dynamics may be realized in a 
manner consistent with First, one can apply a change of frame to unbiased diffusion on 
a ring, and then discretize: clearly, this yields a reversible dynamics. Alternatively, one can 
fashion a time-homogeneous Markov process by coarse-graining the response to a rotating 
potential over one period of its motion. For concreteness, and to keep track of all energy 
fluxes, we consider a dynamics in which the energy function E t (C) is static during each 
timestep, and changed instantaneously between them. As in [iij], the dynamics is assumed 
to satisfy (CD) whilst the potential is static. One can compute transition probabilities over 
the course of one period of rotation of a potential (e.g., a square well) in either direction, 
and show that typically the forward and reverse dynamics, M and M, are not simply related 



to each other, nor do they satisfy (Jo]) [24|]. Due to the coarse-graining, the irreversibility 
measured by AI underestimates the true dissipation in the system. 

We remark that in our framework, coarse-graining generically leads to (and is in fact 



the only mechanism for) the appearance of nonconservative forcing in the system of inter- 
est. By contrast, such forces are central to models based on Langevin equations (see, e.g., 
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251]). are associated with the dissipation of housekeeping heat 16] and have been 
argued to differ fundamentally from those due to a moving potential. Since coarse-graining 
blurs this distinction, it is not clear in what sense it is meaningful. We also remark that 
the nonconservative forces considered in 0,0,0] are assumed not to change under time 
reversal, which even in simple models is not the case for forces due to a moving potential. 
As well as this, it seems often to have been assumed that trajectories are sufficiently detailed 
that the upper bound in (BJ is in fact an equality, and further that housekeeping heat can 
be defined on a per-trajectory basis in terms of the instantaneous state of a system and its 
environment 0, E3] • Such a definition conflicts with the macroscopic quantity described in 



16j if the latter is interpreted as the heat exported by some sequence of dissipative steady 



states if one could somehow switch between them without incurring additional entropy costs. 
For these costs to be removed when averaging over all microscopic realizations of an arbi- 
trary switching process, one finds that details of the history of this process must appear 
in the single-trajectory expressions, contrary to i], 17 1. We thus contend that far greater 
clarity about the meaning of central quantities in the putative framework of steady-state 
thermodynamics 16j is necessary. 

Finally, we examine modeling approaches in which transition probabilities are obtained 



from maximal-entropy inference subject to macroscopic flux constraints [12|. If this is to 
be interpreted as a general recipe for deriving a stochastic dynamics, then we have shown 
the need to derive both the forward and the reverse dynamics, the latter obtained from 
time-reversal of all driving forces, using this procedure. If all macroscopic fluxes simply 
change sign under time reversal, the outcome will be a reversible dynamics, and so — at least 
within the framework put forward here — one needs to argue for time-asymmetric macroscopic 
constraints to realize a dissipative dynamics. However, the theory developed in 12 



is 



intended to apply to internal portions of a larger sheared system, and as such are in contact 
with nonequilibrium reservoirs, not the thermostatted heat baths described here. It would 
be interesting to try and interpret our definition of reversibility in this more general context. 

In summary, we have argued, by examining what it means for a stochastic process to 
be reversible, that the presence of dissipation in a model steady state can only be decided 
once a reverse process, which demands knowledge of the environment, is known. Using 
Jarzynski's detailed fluctuation theorem ([31) and results from information theory, we have 
specified a physical environment that allows information loss to be bounded above by the 
thermodynamic entropy production, extending to a much larger class of nonequilibrium 
systems a result of p|. This allows physical mechanisms by which the system is driven and 
heat dissipated away to be identified in otherwise abstract models of a NESS, illustrating 
with the particular examples shown in Fig. [TJ We note that although the standard detailed 
balance condition ([T]) is satisfied in all these models, only in some is the steady state actually 
dissipative. Although we have couched our discussion in terms of discrete-time Markov 
processes, everything we have said also applies in the continuous-time limit. 

Whilst we have mostly taken a theoretical perspective, we hope that the main result (j3J) 
will be useful experimentally, e.g., to determine whether a stochastic model captures all 



relevant dissipative processes, as we have proposed. The hypothesis that decreases in AI 
under coarse-graining relate to dissipation at a given scale could also be tested explicitly. 
Finally, we see, from the difficulty in discriminating between nonconservative forces and those 
due to coarse-graining a moving potential for example, that in the field of nonequilibrium 
statistical mechanics conceptual problems remain. 
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